Fully Parallel Stochastic Computing Hardware Implementation of Convolutional Neural Networks for Edge Computing Applications.

Edge artificial intelligence (AI) is receiving a tremendous amount of interest from the machine learning community due to the ever-increasing popularization of the Internet of Things (IoT). Unfortunately, the incorporation of AI characteristics to edge computing devices presents the drawbacks of being power and area hungry for typical deep learning techniques such as convolutional neural networks (CNNs). In this work, we propose a power-and-area efficient architecture based on the exploitation of the correlation phenomenon in stochastic computing (SC) systems. The proposed architecture solves the challenges that a CNN implementation with SC (SC-CNN) may present, such as the high resources used in binary-to-stochastic conversion, the inaccuracy produced by undesired correlation between signals, and the complexity of the stochastic maximum function implementation. To prove that our architecture meets the requirements of edge intelligence realization, we embed a fully parallel CNN in a single field-programmable gate array (FPGA) chip. The results obtained showed a better performance than traditional binary logic and other SC implementations. In addition, we performed a full VLSI synthesis of the proposed design, showing that it presents better overall characteristics than other recently published VLSI architectures.

A Noniterative Supervised On-Chip Training Circuitry for Reservoir Computing Systems.

Artificial neural networks (ANNs) is an exponentially growing field, mainly because of its wide range of applications to everyday life such as pattern recognition or time series forecasting. In particular, reservoir computing (RC) arises as an optimal computational framework suited for temporal/sequential data analysis. The direct on-silicon implementation of RCs may help to minimize power and maximize processing speed, that is especially relevant in edge intelligence applications where energy storage is considerably restricted. Nevertheless, most of the RC hardware solutions present in the literature perform the training process off-chip at the server level, thus increasing processing time and overall power dissipation. Some studies integrate both learning and inference on the same chip, although these works are normally oriented to implement unsupervised learning (UL) with a lower expected accuracy than supervised learning (SL), or propose iterative solutions (with a subsequent higher power consumption). Therefore, the integration of RC systems including both inference and a fast noniterative SL method is still an incipient field. In this article, we propose a noniterative SL methodology for RC systems that can be implemented on hardware either sequentially or fully parallel. The proposal presents a considerable advantage in terms of energy efficiency (EE) and processing speed if compared to traditional off-chip methods. In order to prove the validity of the model, a cyclic echo state NN with on-chip learning capabilities for time series prediction has been implemented and tested in a field-programmable gate array (FPGA). Also, a low-cost audio processing method is proposed that may be used to optimize the sound preprocessing steps.

Compact Hardware Synthesis of Stochastic Spiking Neural Networks.

Spiking neural networks (SNN) are able to emulate real neural behavior with high confidence due to their bio-inspired nature. Many designs have been proposed for the implementation of SNN in hardware, although the realization of high-density and biologically-inspired SNN is currently a complex challenge of high scientific and technical interest. In this work, we propose a compact digital design for the implementation of high-volume SNN that considers the intrinsic stochastic processes present in biological neurons and enables high-density hardware implementation. The proposed stochastic SNN model (SSNN) is compared with previous SSNN models, achieving a higher processing speed. We also show how the proposed model can be scaled to high-volume neural networks trained by using back propagation and applied to a pattern classification task. The proposed model achieves better results compared with other recently-published SNN models configured with unsupervised STDP learning.

A Bayesian Target Predictor Method based on Molecular Pairing Energies estimation.

Virtual screening (VS) is applied in the early drug discovery phases for the quick inspection of huge molecular databases to identify those compounds that most likely bind to a given drug target. In this context, there is the necessity of the use of compact molecular models for database screening and precise target prediction in reasonable times. In this work we present a new compact energy-based model that is tested for its application to Virtual Screening and target prediction. The model can be used to quickly identify active compounds in huge databases based on the estimation of the molecule's pairing energies. The greatest molecular polar regions along with its geometrical distribution are considered by using a short set of smart energy vectors. The model is tested using similarity searches within the Directory of Useful Decoys (DUD) database. The results obtained are considerably better than previously published models. As a Target prediction methodology we propose the use of a Bayesian Classifier that uses a combination of different active compounds to build an energy-dependent probability distribution function for each target.

A surface site interaction point methodology for macromolecules and huge molecular databases.

Determining the position and magnitude of Surface Site Interaction Points (SSIP) is a useful technique for understanding intermolecular interactions. SSIPs have been used for the prediction of solvation properties and for virtual co-crystal screening. To determine the SSIPs for a molecule, the Molecular Electrostatic Potential Surface (MEPS) is first calculated using ab initio methods such as Density Functional Theory. This leads to a high cost in terms of computation time and is not compatible with the analysis of huge molecular databases. Herein, we present a method for the fast estimation of SSIPs, which is based on the MEPS calculated from MMFF94 atomic partial charges. The results show that this method can be used to calculate SSIPs for large molecular databases with a much higher speed than the original ab initio methodology. © 2017 Wiley Periodicals, Inc.

FPGA-Based Stochastic Echo State Networks for Time-Series Forecasting.

Hardware implementation of artificial neural networks (ANNs) allows exploiting the inherent parallelism of these systems. Nevertheless, they require a large amount of resources in terms of area and power dissipation. Recently, Reservoir Computing (RC) has arisen as a strategic technique to design recurrent neural networks (RNNs) with simple learning capabilities. In this work, we show a new approach to implement RC systems with digital gates. The proposed method is based on the use of probabilistic computing concepts to reduce the hardware required to implement different arithmetic operations. The result is the development of a highly functional system with low hardware resources. The presented methodology is applied to chaotic time-series forecasting.

High-Density Liquid-State Machine Circuitry for Time-Series Forecasting.

Spiking neural networks (SNN) are the last neural network generation that try to mimic the real behavior of biological neurons. Although most research in this area is done through software applications, it is in hardware implementations in which the intrinsic parallelism of these computing systems are more efficiently exploited. Liquid state machines (LSM) have arisen as a strategic technique to implement recurrent designs of SNN with a simple learning methodology. In this work, we show a new low-cost methodology to implement high-density LSM by using Boolean gates. The proposed method is based on the use of probabilistic computing concepts to reduce hardware requirements, thus considerably increasing the neuron count per chip. The result is a highly functional system that is applied to high-speed time series forecasting.

A New Stochastic Computing Methodology for Efficient Neural Network Implementation.

This paper presents a new methodology for the hardware implementation of neural networks (NNs) based on probabilistic laws. The proposed encoding scheme circumvents the limitations of classical stochastic computing (based on unipolar or bipolar encoding) extending the representation range to any real number using the ratio of two bipolar-encoded pulsed signals. Furthermore, the novel approach presents practically a total noise-immunity capability due to its specific codification. We introduce different designs for building the fundamental blocks needed to implement NNs. The validity of the present approach is demonstrated through a regression and a pattern recognition task. The low cost of the methodology in terms of hardware, along with its capacity to implement complex mathematical functions (such as the hyperbolic tangent), allows its use for building highly reliable systems and parallel computing.

Ultra-fast data-mining hardware architecture based on stochastic computing.

Minimal hardware implementations able to cope with the processing of large amounts of data in reasonable times are highly desired in our information-driven society. In this work we review the application of stochastic computing to probabilistic-based pattern-recognition analysis of huge database sets. The proposed technique consists in the hardware implementation of a parallel architecture implementing a similarity search of data with respect to different pre-stored categories. We design pulse-based stochastic-logic blocks to obtain an efficient pattern recognition system. The proposed architecture speeds up the screening process of huge databases by a factor of 7 when compared to a conventional digital implementation using the same hardware area.

Studying the role of synchronized and chaotic spiking neural ensembles in neural information processing.

The brain is characterized by performing many diverse processing tasks ranging from elaborate processes such as pattern recognition, memory or decision making to more simple functionalities such as linear filtering in image processing. Understanding the mechanisms by which the brain is able to produce such a different range of cortical operations remains a fundamental problem in neuroscience. Here we show a study about which processes are related to chaotic and synchronized states based on the study of in-silico implementation of Stochastic Spiking Neural Networks (SSNN). The measurements obtained reveal that chaotic neural ensembles are excellent transmission and convolution systems since mutual information between signals is minimized. At the same time, synchronized cells (that can be understood as ordered states of the brain) can be associated to more complex nonlinear computations. In this sense, we experimentally show that complex and quick pattern recognition processes arise when both synchronized and chaotic states are mixed. These measurements are in accordance with in vivo observations related to the role of neural synchrony in pattern recognition and to the speed of the real biological process. We also suggest that the high-level adaptive mechanisms of the brain that are the Hebbian and non-Hebbian learning rules can be understood as processes devoted to generate the appropriate clustering of both synchronized and chaotic ensembles. The measurements obtained from the hardware implementation of different types of neural systems suggest that the brain processing can be governed by the superposition of these two complementary states with complementary functionalities (nonlinear processing for synchronized states and information convolution and parallelization for chaotic).

Hardware implementation of stochastic spiking neural networks.

Spiking Neural Networks, the last generation of Artificial Neural Networks, are characterized by its bio-inspired nature and by a higher computational capacity with respect to other neural models. In real biological neurons, stochastic processes represent an important mechanism of neural behavior and are responsible of its special arithmetic capabilities. In this work we present a simple hardware implementation of spiking neurons that considers this probabilistic nature. The advantage of the proposed implementation is that it is fully digital and therefore can be massively implemented in Field Programmable Gate Arrays. The high computational capabilities of the proposed model are demonstrated by the study of both feed-forward and recurrent networks that are able to implement high-speed signal filtering and to solve complex systems of linear equations.

Chaos-based mixed signal implementation of spiking neurons.

A new design of Spiking Neural Networks is proposed and fabricated using a 0.35 microm CMOS technology. The architecture is based on the use of both digital and analog circuitry. The digital circuitry is dedicated to the inter-neuron communication while the analog part implements the internal non-linear behavior associated to spiking neurons. The main advantages of the proposed system are the small area of integration with respect to digital solutions, its implementation using a standard CMOS process only and the reliability of the inter-neuron communication.